Image encoding method and image encoder, image decoding method and image decoder, and image processing system

ABSTRACT

A method and an apparatus improving the coding efficiency by representing encoded data in a compact form by means of a pattern matching technique. A waveform vector similar to a partial signal waveform of an image signal in a predetermined group of waveform vectors is searched for. When a compression encoding is performed for information to identify the waveform vector searched for, similarity information, and a position in an image signal of the partial signal waveform in accordance with a predetermined encoding rule, the position in the image signal of the partial signal waveform is encoded through replacement with position information in a predetermined partial domain of the image signal. As a result; since the position in the image of the partial signal waveform that is the most similar to the waveform vector is encoded through replacement with the position information in the predetermined partial domain of the image signal, it is possible to represent the position more compactly.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Divisional of U.S. patent application Ser. No.10/297,488, filed Dec. 9, 2002, which is a National Stage ofInternational Application PCT/JP02/03498, filed Apr. 8, 2002, and claimspriority to Japanese Patent Application No. 2001-110663, filed Apr. 9,2001.

BACKGROUND OF THE INVENTION

The present invention relates to an image encoding and decoding methodfor encoding a source image such as a moving image and a still image anddecoding the encoded image and, more particularly, to an image encodingmethod, an image encoder, an image decoding method and an image decoderthat can encode an image and decode the encoded image with no use of DCT(discrete cosine transform) employed in standard algorithms of MPEGseries and so on.

Furthermore, the present invention relates to an image processing systemthat encodes and decodes an image in accordance with such an imageencoding method and an image decoding method.

In a conventional standard moving image encoding method such as ITU-TH.26× and MPEG series, DCT (discrete cosine transform) is used as ameans to decrease spatial redundancy. Generally, when an image signal isrepresented in a spatial frequency domain, electric power has a tendencyto concentrate at a low frequency level. The DCT performs an orthogonaltransform for a block formed of 8×8 pixels in an image signal space,decomposes an image signal of a source image into a predeterminedcombination of bases, and obtains coefficients of the bases. The DCT hasa characteristic of increasing the coefficient values, that is, a degreeof bias with respect to a frequency component. Since the DCT especiallyconcentrates the bias on a low frequency level that plays an importantrole in vision, the DCT can enhance compression efficiency by performingan adaptive bit distribution.

On the other hand, when encoding is performed at an extremely low bitrate, a resulting coarse quantization degrades a reconstruction of thecoefficients. Consequently, there arise some problems in that it isimpossible to reconstruct important bases to an intrinsic signalrepresentation. Also, since the DCT operates a closed process on an 8×8image block, the DCT has the tendency that a distortion caused byquantization noticeably appears in a boundary of blocks. That generatesa block distortion and exhibits in the image an element that theoriginal signal does not contain visually, whereby the element isrecognized as a seriously noticeable noise.

A large number of bases are required to faithfully reconstruct a steepluminance fluctuation such as a step edge and a portion of waveformhaving a random pattern. In general, when a weight with respect tovision is considered, a code assignment for a coefficient correspondingto a high frequency level is weighted less than a low frequency level.As a result, the coefficient in the high frequency, which plays animportant role in reconstructing the waveform, is lost. The loss of thecoefficient causes harmful noise peculiar to the DCT and results inimage quality degradation.

In order to overcome such a problem that the DCT entails in a highcompression, a method such that a code representation thereof is freefrom a block structure is proposed. For example, the paper “Very LowBit-rate Video Coding Based on Matching Pursuits” (R.Neff et.al, IEEETrans. on CSVT, vol. 7, pp. 158-171, Feb. 1997) discloses that atechnique “Matching Pursuits” (pattern matching) is used to expand aninter frame prediction error signal in a linear combination of anover-complete basis set. In such a technique, since the larger number ofbases (basic signal patterns) is available than the DCT and a unit ofbasis representation is hot limited to a block, it is possible to obtainsuperior image quality with respect to vision at a low rate of encodingcompared to the DCT encoding.

In order to take advantage of the “Matching Pursuits” encodingtechnique, however, the problem that there is a burden on implementationsuch as the number of operations necessary for the encoding side toperform the basis search is pointed out. Also, it is necessary toefficiently represent position information because the searched basismay be located at an arbitrary pixel position on an image plane.

On the other hand, there is an approach that an encoding distortion iseliminated by using hierarchical encoding. SNR Scalability mode (ISO/IEC13818-2) in MGEG-2 and MPEG-4 Fine Granularity Scalability (FGS) mode(ISO/IEC JTC1/SC29/WG11/N3908) follow this approach. Hereinafter, thehierarchical encoding aiming at compensating such an encoding distortionfactor is called “quality hierarchical encoding”. The qualityhierarchical encoding technique is a technique such that an encodingdistortion generated in an encoding picture in a base layer isseparately encoded as an enhance layer and a decoding side sums signalsobtained by decoding individual layers so as to enhance the quality of adecoded image. Regarding the quality hierarchical encoding technique,the necessary number of transmission bits increases by an amount ofencoding data in the enhance layer. However, since it is possible totransmit the semantic content of a picture only in the base layer, thequality hierarchical encoding technique is favorable for a picturetransmission required to accommodate flexibly to a network such as theInternet and a wireless network whose transmission condition (bit rate,packet loss probability, error rate and so on) varies over time.

In the MPEG-4 FGS, since the DCT is further performed for an encodingerror signal in the enhance layer and the resulting coefficients aretransmitted per bit plane, it is possible to transmit a picture in amanner such that the picture quality is gradually improving in thedecoding side as its name suggests. However, the enhance layer stilldepends on the DCT and the DCT block structure, and a distortioncomponent depending upon the block structure, which shows up in anencoding distortion component in the base layer, generates high orderDCT coefficients. As a result, if little information is used in theenhance layer, the encoding does not work efficiently.

SUMMARY OF THE INVENTION

It is a first object of the present invention to provide a method and anapparatus that improve total coding efficiency by compactly representingencoded data by means of the pattern matching technique.

It is a second object of the present invention to provide an imageprocessing system that encodes and decodes an image in accordance withsuch a method.

In order to achieve the above-mentioned first object, as claimed inclaim 1, the present invention is an image encoding method, comprising:a pattern matching step searching for a waveform vector similar to apartial signal waveform of an image signal in a predetermined group ofwaveform vectors; and a waveform pattern encoding step performing acompression encoding for information to identify the waveform vectorsearched for, similarity information to represent similarity between thepartial signal waveform and the waveform vector searched for, and aposition in an image of the partial signal waveform in accordance with apredetermined encoding rule, wherein the image signal is encoded from acombination of the information to identify the waveform vector, thesimilarity information and the position in the image of the partialsignal waveform, and wherein the position in the image of the partialsignal waveform is encoded through replacement with position informationin a predetermined partial domain of the image signal.

In such an image encoding method, since the position in the image of thepartial signal waveform that is the most similar to the waveform vectoris encoded through the replacement with the position information in thepredetermined partial domain of the image signal, it is possible torepresent the position more compactly. In an image encoding process andan image decoding process according to the method, when an order inwhich each predetermined partial domain of the image signal is processedis determined, it is possible to identify a position in the image signalwith respect to such a position representation.

The above-mentioned image signal may be a signal representing a movingimage or a signal representing a still image. Also, the image signal maybe any additional information that is helpful for a process to encode ordecode the signals.

The similarity information representing the similarity between thepartial signal waveform and the waveform vector is not limited as longas the information is a parameter representing the similarity. Forexample, as described in claim 2, the inner product of the partialsignal waveform and the waveform vector may be used.

Additionally, as claimed in claim 2, the present invention is an imageencoding method, comprising: a pattern matching step searching for awaveform vector in a predetermined group of waveform vectors based uponinner products of the waveform vectors and a partial signal waveform ofan image signal; and a waveform pattern encoding step performing acompression encoding for information to identify the waveform vectorsearched for, an inner product value of the partial signal waveform andthe waveform vector searched for, and a position in an image of thepartial signal waveform in accordance with a predetermined encodingrule, wherein in a process in which the pattern matching step and thewaveform pattern encoding step are iterated until a predeterminedcondition is satisfied, whenever encoding information is obtained at thewaveform pattern encoding step, a reconstructed waveform of a partialsignal waveform obtained from the encoding information is subtractedfrom the image signal so as to produce an image signal to be provided tothe pattern matching step, and the image signal is encoded according tothe encoding information obtained at the waveform pattern step until thepredetermined condition is satisfied, and wherein the position in theimage of the partial signal waveform is encoded through replacement withposition information in a predetermined partial domain of the imagesignal.

When the number of points at which it is determined whether or not thepartial waveform signal and the waveform vector match are decreased, itbecomes possible to reduce the number of operations. From the viewpoint,as claimed in claim 3, in the above-mentioned pattern matching step,pixels formed of a predetermined power of 2×2 of an image signalrepresented by a predetermined pixel unit may be set as a unit, and aposition in an image of a partial signal waveform to be encoded may beidentified.

When a normal image characteristic is considered, it becomes possible toencode a position in a comparatively compact form. From the viewpoint,as claimed in claim 4, the predetermined partial domain may be dividedinto unit blocks to identify a position and may be stratified in a blockdomain formed of one unit block or a plurality of unit blocks based uponan inclusive relation, and the position in the image of the partialsignal waveform may be encoded based upon a sequence of bits indicatingwhether or not each block domain of an individual layer includes aposition of a partial signal waveform to obtain an inner product valueof the partial signal waveform and a waveform pattern in a group ofwaveform patterns.

As claimed in claim 5, the image signal to be processed may beinformation representing a source image to be encoded. As claimed inclaim 6, the image signal to be processed may be informationrepresenting a prediction residual image obtained by performing a motioncompensation prediction for a source image to be processed. As claimedin claim 7, the image signal to be processed may be an encodingdistortion signal generated by encoding a source image in accordancewith a predetermined encoding rule. As claimed in claim 8, the imagesignal to be processed may be information representing a predictionresidual image obtained by performing a motion compensation predictionfor an encoding distortion signal generated by encoding a source imagein accordance with a predetermined encoding rule.

As claimed in claim 11, an image encoder according to theabove-mentioned image encoding method, comprises: a pattern matchingpart searching for a waveform vector similar to a partial signalwaveform of an image signal in a predetermined group of waveformvectors; and a waveform pattern encoding part performing a compressionencoding for information to identify the waveform vector searched for,similarity information to represent similarity between the partialsignal waveform and the waveform vector searched for, and a position inan image of the partial signal waveform in accordance with apredetermined encoding rule, wherein the image signal is encoded from acombination of the information to identify the waveform vector, thesimilarity information and the position in the image of the partialsignal waveform, and wherein the waveform pattern encoding part encodesthe position in the image of the partial signal waveform throughreplacement with information regarding position in a predeterminedpartial domain of the image signal.

Similarly as claimed in claim 12, an image encoder according to theabove-mentioned image encoding method, comprises: a pattern matchingpart searching for a waveform vector in a predetermined group ofwaveform vectors based upon inner products of the waveform vectors and apartial signal waveform of an image signal; and a waveform patternencoding part performing a compression encoding for information toidentify the searched waveform vector, an inner product value of thepartial signal waveform and the waveform vector searched for, and aposition in an image of the partial signal waveform in accordance with apredetermined encoding rule, wherein in a process in which operations onthe pattern matching part and the waveform pattern encoding part areiterated until a predetermined condition is satisfied, whenever encodinginformation is obtained at the waveform pattern encoding part, areconstructed waveform of a partial signal waveform obtained from theencoding information is subtracted from the image signal so as toproduce an image signal to be provided to the pattern matching part, andthe image signal is encoded according to the encoding informationobtained by the waveform pattern part until the predetermined conditionis satisfied, and wherein the position in the image of the partialsignal waveform is encoded through replacement with position informationin a predetermined partial domain of the image signal.

In order to achieve the above-mentioned first object, as claimed inclaim 15, the present invention is an image encoding method, comprising:a first image encoding step performing a compression encoding for afirst image signal per predetermined partial domain in accordance with afirst encoding rule; and a second image encoding step performing acompression encoding for a second image signal per partial signalwaveform of the second image signal in accordance with a second encodingrule, wherein a signal corresponding to an error signal between a firstimage signal provided to the first encoding step and a signal obtainedby performing a local decoding process for encoding information obtainedat the first image encoding step is set as a second image signalprovided to the second image encoding step, and wherein the second imageencoding step comprises: a pattern matching step searching for awaveform vector similar to a partial signal waveform of the second imagesignal in a predetermined group of waveform vectors; and a waveformpattern encoding step performing a compression encoding for informationto identify the waveform vector searched for, similarity information torepresent similarity between the partial signal waveform and thewaveform vector searched for, and a position in the second image of thepartial signal waveform in accordance with the second encoding rule, andwherein in the pattern matching step, a group of waveform vectors to beused is selected among a plurality of groups of waveform vectors basedon a parameter used to perform the compression encoding at the firstimage encoding step and the position in the second image of the partialwaveform signal.

According to such an image encoding method, a group of waveform vectorsto be used is selected among a plurality of groups of waveform vectorsbased upon the parameter used for performing an encoding at the firstimage encoding step and the position in the second image of the partialwaveform signal. As a result, it is possible to select a group ofwaveform vectors corresponding to a characteristic of each image to beencoded and to encode the image signal more appropriately.

The second image information is a signal corresponding to an errorsignal between a first image signal provided to the first encoding stepand a signal obtained by performing a local decoding process forencoding information obtained at the first image encoding step. As aresult, when the first image information provided to the first imageencoding step is assumed to be a source image, the second imageinformation corresponds to an encoding distortion signal generated byencoding the source image. In such a case, according to the imageencoding method, it is possible to efficiently encode a signalcorresponding to the error signal ending up with unnecessary additionalinformation when the source image is encoded.

Also, the similarity information between the partial signal waveform andthe waveform vector may be replaced with an inner product between them.In this case, as claimed in claim 16, the present invention is an imageencoding method, comprising: a first image encoding step performing acompression encoding for a first image signal per predetermined partialdomain in accordance with a first encoding rule; and a second imageencoding step performing a compression encoding for a second imagesignal per partial signal waveform of the second image signal inaccordance with a second encoding rule, wherein a signal correspondingto an error signal between a first image signal provided to the firstencoding step and a signal obtained by performing a local decodingprocess for encoding information obtained at the first image encodingstep is set as a second image signal provided to the second imageencoding step, and wherein the second image encoding step comprises apattern matching step searching for a waveform vector in a predeterminedgroup of waveform vectors based upon inner products of the waveformvectors and a partial signal waveform of the second image signal; and awaveform pattern encoding step performing a compression encoding forinformation to identify the waveform vector searched for, an innerproduct value of the partial signal waveform and the waveform vectorsearched for, and a position in the second image of the partial signalwaveform in accordance with the second encoding rule, wherein in aprocess in which the pattern matching step and the waveform patternencoding step are iterated until a predetermined condition is satisfied,whenever encoding information is obtained at the waveform patternencoding step, a reconstructed waveform of a partial signal waveformobtained from the encoding information is subtracted from the secondimage signal so as to produce a second image signal to be provided tothe pattern matching step, and the second image signal is encodedaccording to the encoding information obtained at the waveform patternstep until the predetermined condition is satisfied, and wherein theposition in the image of the partial signal waveform is encoded throughreplacement with position information in a predetermined partial domainof the image signal and in the pattern matching step, a group ofwaveform vectors to be used is selected among a plurality of groups ofwaveform vectors based on a parameter used to perform the compressionencoding at the first image encoding step and the position in the secondimage of the partial waveform signal.

As claimed in claim 23, an image encoder that operates a process inaccordance with the above-mentioned image encoding method, comprises: afirst image encoding part performing a compression encoding for a firstimage signal per predetermined partial domain in accordance with a firstencoding rule; and a second image encoding part performing a compressionencoding for a second image signal per partial signal waveform of thesecond image signal in accordance with a second encoding rule, wherein asignal corresponding to an error signal between a first image signalprovided to the first encoding part and a signal obtained by performinga local decoding process for encoding information obtained by the firstimage encoding part is set as a second image signal provided to thesecond image encoding part, and wherein the second image encoding partcomprises: a pattern matching part searching for a waveform vectorsimilar to a partial signal waveform of the second image signal in apredetermined group of waveform vectors; and a waveform pattern encodingpart performing a compression encoding for information to identify thewaveform vector searched for, similarity information to representsimilarity between the partial signal waveform and the waveform vectorsearched for, and a position in the second image of the partial signalwaveform in accordance with the second encoding rule, and wherein asecond image signal is encoded based upon a combination of theinformation to identify the waveform vector, the similarity informationand the position in the second image of the partial signal waveform, andwherein the pattern matching part comprises a plurality of groups ofwaveform vectors; and a waveform vector group selection part selecting agroup of waveform vectors to be used among the groups of waveformvectors based upon a parameter used by the first image encoding part toperform the compression encoding and the position in the second image ofthe partial waveform signal.

Similarly, as claimed in claim 24, the present invention is an imageencoder, comprising: a first image encoding part performing acompression encoding for a first image signal per predetermined partialdomain in accordance with a first encoding rule; and a second imageencoding part performing a compression encoding for a second imagesignal per partial signal waveform of the second image signal inaccordance with a second encoding rule, wherein a signal correspondingto an error signal between a first image signal provided to the firstencoding part and a signal obtained by performing a local decodingprocess for encoding information obtained by the first image encodingpart is set as a second image signal provided to the second imageencoding part, and wherein the second image encoding part comprises apattern matching part searching for a waveform vector in a predeterminedgroup of waveform vectors based upon inner products of the waveformvectors and a partial signal waveform of the second image signal; and awaveform pattern encoding part performing a compression encoding forinformation to identify the waveform vector searched for, an innerproduct value of the partial signal waveform and the waveform vectorsearched for, and a position in the second image of the partial signalwaveform in accordance with the second encoding rule, wherein in aprocess in which operations on the pattern matching part and thewaveform pattern encoding part are iterated until a predeterminedcondition is satisfied, whenever encoding information is obtained by thewaveform pattern encoding part, a reconstructed waveform of a partialsignal waveform obtained from the encoding information is subtractedfrom the second image signal so as to produce a second image signal tobe provided to the pattern matching part, and the second image signal isencoded according to the encoding information obtained by the waveformpattern part until the predetermined condition is satisfied, and whereinthe pattern matching part comprises a plurality of groups of waveformvectors; and a waveform vector group selection part selecting a group ofwaveform vectors to be used among the groups of waveform vectors basedupon a parameter used by the first image encoding part to perform thecompression encoding and the position in the second image of the partialwaveform signal.

In order to achieve the above-mentioned first object, as claimed inclaim 29, the present invention is an image decoding method forreceiving compressed image information and reconstructing imageinformation by decompressing the compressed image information perpredetermined partial domain, the image decoding method comprising thesteps of: decoding compressed image information regarding apredetermined partial domain in accordance with a predetermined decodingrule and obtaining information to identify a waveform vector, similarityinformation to represent similarity between a partial signal waveformand the waveform vector, and a position in an image of the partialsignal waveform; reconstructing image information based upon thewaveform vector identified from a predetermined group of waveformvectors by the information to identify a waveform vector, the similarityinformation, and the position in the image of the partial signalwaveform; and decoding the position in the image of the partial signalwaveform included in the compressed image information as information perpredetermined partial image domain when the compressed image informationis decoded.

Additionally, as claimed in claim 30, the present invention is an imagedecoding method for receiving compressed image information andreconstructing image information by decompressing the compressed imageinformation per predetermined partial domain, the image decoding methodcomprising the steps of: decoding compressed image information regardinga predetermined partial domain in accordance with a predetermineddecoding rule and obtaining information to identify a waveform vector,an inner product value of the waveform vector and a partial signalwaveform, and a position in an image of the partial signal waveform;reconstructing image information based upon the waveform vectoridentified from a predetermined group of waveform vectors by theinformation to identify a waveform vector, the inner product value, andthe position in the image of the partial signal waveform; and decodingthe position in the image of the partial signal waveform included in thecompressed image information as information per predetermined partialimage domain when the compressed image information is decoded.

According to the above-mentioned image decoding methods, when a positionin an image of a partial signal waveform is decoded through replacementwith position information in a predetermined partial domain of the imagesignal, it is possible to reconstruct the position in the image of thepartial signal waveform.

As claimed in claim 38, an image decoder that operates a process inaccordance with the above-mentioned image decoding method, for receivingcompressed image information and reconstructing image information bydecompressing the compressed image information per predetermined partialdomain, comprises: a first part decoding compressed image informationregarding a predetermined partial domain in accordance with apredetermined decoding rule and obtaining information to identify awaveform vector, similarity information to. represent similarity betweena partial signal waveform and the waveform vector, and a position in animage of the partial signal waveform; and a second part reconstructingimage information based upon the waveform vector identified from apredetermined group of waveform vectors by the information to identify awaveform vector, the similarity information, and the position in theimage of the partial signal waveform, wherein the first part decodes theposition in the image of the partial signal waveform included in thecompressed image information as information per predetermined partialimage domain when the compressed image information is decoded.

Additionally, as claimed in claim 39, the present invention is an imagedecoder for receiving compressed image information and reconstructingimage information by decompressing the compressed image information perpredetermined partial domain, comprising: a first part decodingcompressed image information regarding a predetermined partial domain inaccordance with a predetermined decoding rule and obtaining informationto identify a waveform vector, an inner product value of the waveformvector and a partial signal waveform, and a position in an image of thepartial signal waveform; and a second part reconstructing imageinformation based upon the waveform vector identified by the informationto identify the waveform vector from a predetermined group of waveformvectors, the inner product value, and the position in the image of thepartial signal waveform, wherein the first part decodes the position inthe image of the partial signal waveform included in the compressedimage information as information per predetermined partial image domainwhen the compressed image information is decoded.

Additionally, as claimed in claim 47, the present invention is an imagedecoding method, comprising: a first image decoding step receiving afirst compressed image and reconstructing a first image information bydecompressing the first compressed image information per predeterminedpartial domain; a second image decoding step receiving a secondcompressed image and reconstructing a second image information bydecompressing the second compressed image information per predeterminedpartial signal waveform; and an image synthesizing step obtaining outputimage information by synthesizing the first image information and thesecond image information, wherein in the second image decoding step, bydecoding the second compressed image information in accordance with apredetermined decoding rule, information to identify a waveform vector,similarity information to represent similarity between a partial signalwaveform and the waveform vector, and a position in an image of thepartial signal waveform are obtained, and a group of waveform vectors tobe used is selected among a plurality of predetermined groups ofwaveform vectors based upon a code parameter included in the firstcompressed image information provided to the first image decoding stepand the position in the image of the partial waveform signal, andwherein the second image information is generated based upon thewaveform vector identified by the information to identify the waveformvector in the selected group of waveform vectors, the similarityinformation and the position in the image of the partial signalwaveform.

Additionally, as claimed in claim 48, the present invention is an imagedecoding method, comprising: a first image decoding step receiving afirst compressed image and reconstructing a first image information bydecompressing the first compressed image information per predeterminedpartial domain; a second image decoding step receiving a secondcompressed image and reconstructing a second image information bydecompressing the second compressed image information per predeterminedpartial signal waveform; and an image synthesizing step obtaining outputimage information by synthesizing the first image information and thesecond image information, wherein in the second image decoding step, bydecoding the second compressed image information in accordance with apredetermined decoding rule, information to identify a waveform vector,an inner product value of the waveform vector and a partial signalwaveform, and a position in an image of the partial signal waveform areobtained, and a group of waveform vectors to be used is selected among aplurality of predetermined groups of waveform vectors based upon a codeparameter included in the first compressed image information provided tothe first image decoding step and the position in the image of thepartial waveform signal, and wherein the second image information isgenerated based upon the waveform vector identified by the informationto identify the waveform vector in the selected group of waveformvectors, the inner product value and the position in the image of thepartial signal waveform.

As claimed in claim 55, an image decoder that operates a process inaccordance with the above-mentioned image decoding method, comprises: afirst image decoding part receiving a first compressed image andreconstructing a first image information by decompressing the firstcompressed image information per predetermined partial domain; a secondimage decoding part receiving a second compressed image andreconstructing a second image information by decompressing the secondcompressed image information per predetermined partial signal waveform;and an image synthesizing part obtaining output image information bysynthesizing the first image information and the second imageinformation, wherein the second image decoding part comprises aplurality of predetermined groups of waveform vectors; a first partdecoding the second compressed image information in accordance with apredetermined decoding rule and obtaining information to identify awaveform vector, similarity information to represent similarity betweena partial signal waveform and the waveform vector, and a position in animage of the partial signal waveform; a second part selecting a group ofwaveform vectors to be used among a plurality of predetermined groups ofwaveform vectors based upon a code parameter included in the firstcompressed image information provided to the first image decoding partand the position in the image of the partial waveform signal; and athird part generating the second image information based upon thewaveform vector identified by the information to identify the waveformvector from the selected group of waveform vectors, the similarityinformation and the position in the image of the partial signalwaveform.

As claimed in claim 56, the present invention is an image decoder,comprising: a first image decoding part receiving a first compressedimage and reconstructing a first image information by decompressing thefirst compressed image information per predetermined partial domain; asecond image decoding part receiving a second compressed image andreconstructing a second image information by decompressing the secondcompressed image information per predetermined partial signal waveform;and an image synthesizing part obtaining output image information bysynthesizing the first image information and the second imageinformation, wherein the second image decoding part comprises aplurality of predetermined groups of waveform vectors; a first partdecoding the second compressed image information in accordance with apredetermined decoding rule and obtaining information to identify awaveform vector, an inner product value of a partial signal waveform andthe waveform vector, and a position in an image of the partial signalwaveform; a second part selecting a group of waveform vectors to be usedamong a plurality of predetermined groups of waveform vectors based upona code parameter included in the first compressed image informationprovided to the first image decoding part and the position in the imageof the partial waveform signal; and a third part generating the secondimage information based upon the waveform vector identified by theinformation to identify the waveform vector from the selected group ofwaveform vectors, the inner product value, and the position in the imageof the partial signal waveform.

In order to achieve the above-mentioned second object, as claimed inclaims 63 and 64, there is provided an image processing systemscomprising combinations of the above-mentioned image encoder and imagedecoder.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example of a structure of animage encoding and decoding system according to a first embodiment ofthe present invention;

FIG. 2 is a flowchart illustrating an example of a procedure for animage encoding process;

FIG. 3 is a flowchart illustrating an example of a procedure for animage decoding process;

FIG. 4 is a flowchart illustrating an example of a procedure for aprediction image generating process;

FIG. 5 is a block diagram illustrating a structure of a basis encodingpart;

FIG. 6 is a diagram illustrating an example of a method for identifyinga position of a basis search point in a macroblock;

FIG. 7 is a diagram illustrating an example of position information ofthe basis search point identified in accordance with the method shown inFIG. 6;

FIG. 8 is a block diagram illustrating an example of a structure of animage encoding and decoding system according to a second embodiment ofthe present invention;

FIG. 9 is a flowchart illustrating an example of a procedure for animage encoding process;

FIG. 10 is a flowchart illustrating an example of a procedure for animage decoding process;

FIG. 11 is a block diagram illustrating an example of a structure of animage encoding and decoding system according to a third embodiment ofthe present invention;

FIG. 12 is a block diagram illustrating an example of a structure of animage encoding and decoding system according to a fourth embodiment ofthe present invention;

FIG. 13 is a flowchart illustrating an example of a procedure for animage encoding process; and

FIG. 14 is a block diagram illustrating a variation of an enhance-layerencoding part and an enhance-layer decoding part.

DETAILED DESCRIPTION OF THE INVENTION

In the following, embodiments of the present invention will be describedwith reference to the accompanying drawings.

For example, an image encoder and an image decoder according to thefirst embodiment of the present invention are constructed as is shown inFIG. 1. In the example, the “Matching Pursuits” (pattern matching)technique is used to perform a single layer picture encoding anddecoding.

In FIG. 1, an image encoder 2 comprises an encoding control part 4, arate buffer 5, a prediction signal generating part 7-1, a frame memory11-1, a motion detecting part 12, a basis searching part 13, a basiscode book 14-1, a basis encoding part 15, a basis local decoding part 16and a filter 23-1. Also, an image decoder 17, which is connected withthe image encoder 2 via a predetermined line 30, comprises a decodingcontrol part 18, a receiving buffer 19, a basis decoding part 21, adisplaying buffer 22, a filter 23-2, a frame memory 11-2, a predictionsignal generating-part 7-2 and a basis code book 14-2.

In the following, a description will be given of an operation on theimage decoder 2 (encoding procedure).

The image encoder 2 performs a compression encoding for an individualframe of a picture signal 1, which is considered to be a unit.Furthermore, each frame is encoded per macroblock formed of 16 pixels X16 lines of a luminance signal. The encoded information is multiplexedto a bitstream 3 and is transmitted to the image decoder 17 via the line30.

For example, the encoding process is performed in accordance with aprocedure shown in FIG. 2.

In FIG. 2, the encoding control part 4 sets initial parameters necessaryfor performing an encoding together with an acceptable code amount ofthe frame of interest (initialization: step S1). The code parametersinclude an initial quantization parameter and encoding mode informationindicating whether or not the frame is encoded exclusively in the frameor whether or not the frame is encoded by using an inter frameprediction. In this embodiment of the present invention, a code amountin the frame mainly depends on the number of pieces of basis informationto be encoded (hereinafter, referred to as an atom). The reason will bementioned later. After the second frame, the acceptable code amount inthe next frame is set by receiving a feedback of an occupied amount 6 tothe encoding control part 4 so that the rate buffer 5 cannot overflow.

Next, the prediction signal generating part 7-1 generates a predictionimage 9 to obtain a prediction residual signal 8 to be encoded (generateprediction image: step S2). The prediction image 9 is generated inaccordance with a procedure shown in FIG. 4. The prediction image 9 isdefined per macroblock.

In FIG. 4, a macroblock counter n is reset (step S2-1). Next, the intraframe prediction is performed for the macroblock n (step S2-2). Avariety of intra frame prediction methods is assumed. For example, apixel average of an input macroblock (DC component) may be used as aprediction image. Also, values of marginal pixels of the macroblock thathave already been encoded are buffered, and the values may be used togenerate a prediction image through a plane prediction. In step S2-2, aprediction cost incurred by the prediction image data (P_intra) and theintra frame prediction is computed. When a plurality of predictionmethods are used, intra frame prediction mode information is alsoproduced to indicate what type of intra frame prediction method has beenused. Regarding the prediction cost, a sum of absolute difference (SAD)and a mean square error (MSE) may be used to represent residual power.Besides, a code amount-distortion cost representation includingadditional information required for an adopted prediction method (suchas above-mentioned intra frame prediction mode information) may be used.

Then, an inter frame prediction is performed by a motion compensationprediction (step S2-3). The inter frame prediction identifies a similarimage in reference images in the frame memory 11 by predicting a motionbetween frames and sets the identified similar image as a predictionimage. The motion detecting part 12 detects a motion vector asinformation to identify the similar image in the reference images. Themotion vector may be represented in any form such as a parallel shiftvector by block matching often used in existing picture encodingmethods, a vector representing an affine model, or a vector representinga perspective transform model. In general, the more complicated a modelis and the larger are the number of motions that can be represented, thesmaller the prediction residual is. On the other hand, since arithmeticoperations for the motion detection and the number of parameters for themodel representation increase corresponding to the above benefit, it isnecessary to make a selection from the viewpoint of a balance betweenimplementation burden and performance. Regarding motion vector searchand representation method, any representation is available in thisembodiment of the present invention as long as it can be represented asmotion information representation having a macroblock as a unit.

Also, like an 8×8 pixel unit prediction mode of MPEG-4, a macroblock maybe further divided into smaller blocks and a motion vector may be foundfor each divided block. Like a B frame prediction of MPEG-½, a pluralityof frames may be referred to in order to perform a prediction. In a casein which a plurality of prediction methods are switched, inter frameprediction mode information is produced to indicate what type ofprediction method has been used. The inter frame prediction mode may besuch a mode that “Matching Pursuits” encoding data of an encoded frameis saved (for the storage, it is required to prepare a memory space tosave an atom parameter 24 corresponding to the frame memory 11) and atominformation encoded in a reference frame is inherited without change.

At step S2-3, prediction image data (P_inter), a motion vector, interframe prediction mode information, and a prediction cost incurred by amotion compensation prediction are computed. Regarding the predictioncost, the SAD, the MSE and the code amount-distortion cost may be usedlike the intra frame prediction. As a code amount which should beconsidered in the inter frame prediction, there are a code amount of themotion vector itself, switch information of a motion vector model, andinter frame prediction mode information and so on.

At step S2-4, it is determined which type of prediction is performed forthe macroblock of interest, the intra frame prediction (INTRA) or theinter frame prediction (INTER). The two above-mentioned prediction costsare used as an evaluation value with respect to the prediction and theprediction type incurring the smaller prediction cost is chosen. If theintra frame prediction (INTRA) is chosen, a prediction image Pred(n) forthe macroblock n is set to a P_intra and the intra frame prediction modeinformation is output as output information 10 (step S2-5). If the interframe prediction (INTER) is chosen, a prediction image Pred(n) is set toa P_inter and the inter frame prediction mode information together witha motion vector is output as the output information 10 (step S2-6).After that, the macroblock counter n is incremented (S2-7). If theabove-mentioned process is completed for every macroblock in the frame,the prediction image generating process is completed.

Back to FIG. 2, after completing the above-mentioned prediction imagegenerating process, regarding the generated prediction image 9, thefilter 23-1 suppresses discontinuity in a boundary between blocks causedby the motion detection per macroblock. After that, a difference betweenthe resulting prediction image and the input picture frame 1 is computedand becomes the prediction residual signal 8 to be encoded (generateprediction residual signal: step S3). The process on the filter 23-1prevents any component unnecessary in the “Matching Pursuits” encodingprocess from being encoded.

The prediction residual signal 8 generated in the above-mentioned manneris input to the basis searching part 13 and is transformed in the formof a linear combination of basis vectors based upon the “MatchingPursuits” technique (step S4). The detailed description of a movingimage encoding based upon the “Matching Pursuits” is referred to in“Very Low Bit-rate Video Coding Based on Matching Pursuits” (R.Neffet.al, IEEE Trans. on CSVT, vol. 7, pp. 158-171, Feb. 1997). Regardingan extraction process of encoded data, this embodiment of the presentinvention follows a process mentioned in the above reference. Theembodiment of the present invention has a characteristic in that theembodiment is designed to efficiently represent the extracted encodingdata by using a small amount of code.

The “Matching Pursuits” technique in which a signal f is represented inthe form of a linear combination of a set gk of predeterminedover-complete basis functions (waveform patterns) searches for a basisgk such that an inner product of the gk and a signal Rnf (a signal to berepresented as a basis at the n searching step: a partial signalwaveform) is maximized. As is shown in the formula (1), the signal f isrepresented as a linear combination such that the linear combination hasas its coefficients the inner values $\begin{matrix}{\left\langle {{R_{n}f},g_{k_{n}}} \right\rangle.} & \quad \\{{f = {\sum\limits_{n = 0}^{m - 1}{< {R_{n}f}}}},{g_{kn} > {g_{kn} + {R_{m}f}}},} & (1)\end{matrix}$where n is the number of basis searching steps, gk is the basis found atthe n-th searching step such that the inner product

(Rn^(f),g_(k) _(n) )

is maximized, and Rmf is a residual component to be searched for at them-th searching step. Thus, as the number of steps n increases, therepresentation accuracy of the signal f is enhanced. That means that thelarger the number of bases used to represent the signal f, the moreaccurately the signal f can be represented. Therefore, the larger thenumber of bases is, that is, the more times of searching steps thereare, the more the code amount is and the smaller the distortion is.

An ideal basis search corresponds to a process for searching for a basissuch that for every pixel in a frame, the above inner product ismaximized among all predetermined bases (waveform patterns) with respectto a signal waveform whose center is the pixel. For that search, it isnecessary to compute enormous arithmetic operations (the number ofpixels in a frame)×(the number of bases) of the inner product. In orderto decrease the number of arithmetic operations without the loss ofoptimality as much as possible, the above-mentioned reference proposes amethod to detect a domain with high electric power and search for abasis in an area adjacent to the domain.

FIG. 5 shows a diagram illustrating a detailed structure of the basissearching part 13 according to the embodiment of the present invention.

In FIG. 5, the basis searching part 13 comprises a search start pointdetermination part 13-1 and a basis determination part 13-3. The searchstart point determination part 13-1 receives the prediction residualsignal 8 as its input and searches for a domain with the maximalelectric power in the prediction residual signal 8. A position in thesearched domain is considered as a start point of the basis search, andimage data (partial signal waveform) of an area adjacent to the startpoint (for example, S pixels×S lines) is supplied to the basisdetermination part 13-3 as an inner product arithmetic target signal(Rnf) 13-2. The basis determination part 13-3 retrieves individual basiscode words 13-4 from the basis code book 14, performs inner productoperations of the inner product arithmetic target signal 13-2 and theindividual basis code words 13-4, and determines a pair of signal andbasis such that the inner product is maximized. Regarding the predictionresidual signal 8, the basis gk is determined among basis vectors in thebasis code book 14. Then, the basis determination part 13-3 outputs as acode parameter 24 a code number (an index) of the basis gk, the innerproduct (basis coefficient) and a pixel position pk=(xk, yk) where thebasis gk is applied. Here, the pixel position pn corresponds to aposition of the center pixel of Rnf. A triple of the code parameters isreferred to as an atom.

The basis determination part 13-3 supplies a signal reconstructed fromthe atom in accordance with the above formula (1). The reconstructedsignal is subtracted from the prediction residual signal 8, and then theprediction residual signal is updated for the next searching step.Hereinafter, the searching step is incremented until the acceptable codeamount is reached. The atom is determined for each of the searchingsteps.

In order to reconstruct the signal f by the image decoder 17, it isnecessary to encode and transmit the atom parameter 24 extracted foreach searching step. Back to FIG. 2, the basis encoding part 15 servesfor this encoding operation (step S5). A basis index of the atomundergoes a compression encoding operation by means of a variable-lengthencoding corresponding to emergence frequency of the basis index, and abasis coefficient of the atom undergoes a compression encoding operationby means of a scalar quantization. (It is noted that the basiscoefficient is quantized in the basis determination part 13-3 in FIG. 5because the basis coefficient is required to represent a signal for eachsearching step and gives an influence to the basis searching process.)Furthermore, it is necessary to encode the position information pn ofthe atom. If the atom is supposed to be located at an arbitrary pixelposition in the frame, there arises a problem in that a code amountrequired to represent the position increases.

Thus, in the embodiment of the present invention, feasable values to theposition pn are restricted to multiplicative values of a 2×2 block. As aresult, since the center of an atom is always located at a vertex of the2×2 block, it can be determined per 2×2 block whether or not there is anatom to be encoded. Since the image encoder 2 and the image decoder 17according to the embodiment of the present invention have a bitstreamformed of a macroblock with 16×16 pixels as a unit, the informationregarding whether or not the atom exists can be described as afour-layer quadrival tree structure whose root is a macroblock as shownin FIG. 6. FIG. 7 shows an example of a code representation of thequadrival tree structure.

In FIG. 7, the level of the macroblock (16×16) represents by 1 bit justwhether there is an atom in an interior of the macroblock (1) or not(0). The level of block (8×8) represents by at most 4 bits how atoms aredistributed in four 8×8 blocks included in the macroblock. Similarly,the level of block (4×4) and the level of block (2×2) have coderepresentations by at most 16 bits and 64 bits, respectively. Here, ifthere appears a vertex having a code “0”, not any code has to beassigned for successors of the vertex. For example, in a case in whichthere is no atom in a macroblock, the quadrival tree may have only onevertex information item. Also, according to a property of the predictionresidual signal, atoms often tend to concentrate spatially. Thus, whenan optimal variable-length code is designed based upon a conditionalprobability and an arithmetic code or the like is employed, it ispossible to shorten an average code length by assigning a short codelength for a tree having many ls.

By applying the above-mentioned method, it is possible to decrease thenumber of basis searching points and the arithmetic operations thereofrather than the method to perform a search operation per pixel. Also,the above-mentioned method represents a position of the basis searchingpoint pn as the position in the macroblock by using a tree structurewhose root is the macroblock. As a result, whatever display size theimage information has, it is possible to fix a maximal code amountrequired to identify the position of the basis searching point.Furthermore, in a domain where atoms do not concentrate, the treestructure makes it possible to represent the position pn of a basissearching point by a lesser code amount and decrease the code amountfrom a whole frame required to represent the position pn of each basissearching point, if general characteristics of an image are considered.

Whenever atoms are encoded for each of the above-mentioned search steps,it is determined at step S8 whether or not the code amount reaches theacceptable code amount. If the code amount reaches the acceptable codeamount, the basis searching process is stopped and the encoding of theprediction residual signal of the frame of interest is terminated. Then,if there remain some frames (step S9), the process for the next frame isperformed back to step S1. If there is no remaining frame, the processis terminated.

When the encoding process for atoms corresponding to one frame iscompleted, a local decoding process is performed and the frame memory 11is updated so as to obtain a reference image for the motion compensationprediction after the next frame (the basis local decoding part 16, stepsS6 and S7).

In the above-mentioned manner, the bitstream 3 including encodinginformation generated by the image encoder 2 is supplied to the imagedecoder 17 via the line 30.

On the other hand, the image decoder 17 receives the bitstream 3including the above encoding information and performs the same processas the basis local decoding part 16. In a case in which an image isdecoded and displayed in real time, the decoding process is performed inaccordance with steps that will be mentioned with respect to FIG. 3.

In FIG. 3, based upon occupied information 20 of bitstream data saved inthe receiving buffer 19, a time management function of the decodingcontrol part 18 determines a decoding start time of a frame being at thetime t and runs a basis decoding part 21 at the time (step S10). Thebasis decoding part 21 detects a frame synchronous word and analyzesatom data of the bitstream in accordance with a predetermined syntaxrule (step S11). The prediction signal generating part 7-2 (whichperforms a process corresponding to the prediction signal generatingpart 7-1 in the image encoder 2) receives prediction result outputinformation 10 including information regarding a motion vector and aprediction mode in the analyzed data, and then generates a predictionimage 9 (step S12). An atom decoding process identifies a basis withreference to a basis index and a basis code book 14-2 equivalent to thebasis code book 14-1 in the image encoder 2 and reconstructs aprediction residual signal f(x, y) based upon the formula (1) (stepS13). The position information (ref. FIG. 6 and FIG. 7) regarding amacroblock where the basis is applied is transformed into a position ina frame of the prediction residual signal.

The prediction residual signal f(x, y) is generated by the predictionsignal generating part 7-2, is processed by a filter 23-2 and is addedas the prediction image 9. Then, a final frame decoding image isgenerated by using the above position in the frame. The decoded image issaved in a displaying buffer 22 (step S14) and is written in the framememory 11 for the decoding process of the next frame. The displayingbuffer 22 may be provided to use a portion of a frame memory 11-2.

At the displaying time that is prescribed in the bitstream or isdetermined according to a rule prescribed in the image decoder 2, thedecoding control part 18 runs the displaying buffer 22 and displays theframe image being at the time t on the screen (steps S15 and S16).

In this embodiment of the present invention, when an atom, which isformed of code parameters, is encoded in accordance with the “MatchingPursuits” method, it is possible to efficiently encode per macroblockthe atom that may be at an arbitrary position in a frame. That improvesthe overall coding efficiency.

Regarding FIG. 1, the line 30 may be a network such as the Internet or adata reading circuit for reading and decoding a bitstream from a mediumin which an output result of the image encoder 2 is recorded.

A description will now be given of an image encoder and an image decoderaccording to a second embodiment of the present invention. Especially inthis embodiment, a description will be given of an example of an imageencoder and an image decoder to perform a quality hierarchical encodingand a transmission by using the “Matching Pursuits”. In the imageencoder and the image decoder according to the embodiment, the “MatchingPursuits” provides a basis representation of an encoding distortionsignal, which is transmitted as additional information. As a result,when a line in use has a good condition, it becomes possible to transmita high quality picture.

FIG. 8 shows a structure of the image encoder and the image decoderaccording to the embodiment of the present invention.

In FIG. 8, the image encoder comprises an encoding control part 111, abase-layer encoding part 102, a base-layer local decoding part 103, abase-layer frame memory 104-1, an enhance-layer encoding part 106 and arate buffer 112. The enhance-layer encoding part 106 for producing andencoding the basis representation of an encoding distortion signalcomprises a basis code book 108-1, a basis searching part 109 and abasis encoding part 110.

On the other hand, the image decoder, which is connected with the imageencoder via the line 30, comprises a decoding control part 114, areceiving buffer 115, a base-layer decoding part 117, a base-layer framememory 104-2, an enhance-layer decoding part 118 and a displaying buffer122. The enhance-layer decoding part 118 for decoding an encodingdistortion signal transmitted from the image encoder as enhance-layerinformation comprises a basis decoding part 119 and a basis code book108-2.

The image encoder performs a compression encoding for a picture signal(a source image) per frame. The image encoder represents a signalunderlying a picture through a base layer encoding and encodes anencoding distortion signal, which is represented as a difference betweenthe source image and the encoded image, in the enhance layer. For eachlayer, individual frames are encoded per macroblock formed of 16pixels×16 lines of a luminance signal. After a base-layer bitstream andan enhance-layer bitstream are generated separately, an eventualbitstream is formed. In general, the image decoder is formed so as toperform an inter layer synthesis (in which a decoded image in theenhance layer, that is, an encoding distortion component, is added to adecoded image in the base layer) by using a time stamp of an individualframe. Since the bitstream in the base layer and the bitstream in theenhance layer are separated, it is possible to perform transmissioncontrol such that at a sending process, only the bitstream in the baselayer is sent to the image decoder that can receive only the base layer.

The image encoder performs an encoding process, for example, inaccordance with a procedure as shown in FIG. 9.

In FIG. 9, the encoding control part 111 sets initial code parametersnecessary to perform an encoding together with an acceptable code amountof the frame of interest (step S1). The code parameters includes aninitial quantization parameter and encoding mode information indicatingwhether or not a frame is encoded exclusively in the frame and whetheror not the frame is encoded by using an inter frame prediction. (It isnoted that in this embodiment of the present invention, the enhancelayer has no inter frame prediction.) The code parameters are determinedseparately for the based layer and the enhance layer based upon an interlayer code amount allocation rule according to the use of an encodingand decoding system. As mentioned in the first embodiment of the presentinvention, an adjusting factor of the code amount in the enhance layeris mainly the number of atoms.

After the second frame, the acceptable code amount of the next frame isset by receiving a feedback of occupied amount 113 to the encodingcontrol part 111 so that the rate buffer 112 cannot overflow.

The base-layer encoding part 102 encodes the picture signal 1 (thesource image) in accordance with an arbitrary picture encoding methodsuch as MPEG-4 video encoding method (ISO/IEC 14496-2) and ITU-T H.263(step S102). Here, the picture signal 1 may be encoded in accordancewith the “Matching Pursuits” method mentioned in the first embodiment ofthe present invention. Encoded data of an individual frame is decodedinto a decoded image 105 by the base-layer local decoding part 103 andis saved in the frame memory 104-1 so as to perform a motioncompensation prediction after the next frame.

A base-layer encoding distortion signal 107 is generated by computing adifference between the input picture signal 1 and the decoded image 105in the base layer (step S103). The enhance-layer encoding part 106encodes the encoding distortion signal 107 based upon the “MatchingPursuits”. A procedure of the “Matching Pursuits” encoding according tothis embodiment (steps S104 through S109) is identical to the procedurementioned in steps S4 through S9 (ref. FIG. 2) according to the firstembodiment. It is noted that the enhance-layer encoding part 106 has noprocessing part serving to execute an inter frame motion compensationprediction process, because the enhance-layer encoding part 106 performsan intra frame encoding for the encoding distortion signal 107. Thebasis searching part 109 and the basis encoding part 110 performprocesses corresponding to the basis searching part 13 and the basisencoding part 15 shown in FIG. 1, respectively.

It is noted that although the basis code book 108-1 has the sameconstitution as the basis code book 14-1 shown in FIG. 1, a basispattern, that is, a code word, may be redesigned to adapt acharacteristic of the encoding distortion signal 107. Especially, whenemploying a conventional picture encoding method in which the DCT isused to encode the base layer, there appears a visually noticeablepattern in the base layer such as a block distortion peculiar to theDCT. The use of the basis code book aiming at such a pattern makes itpossible to efficiently encode the enhance layer.

The image decoder receives a bitstream transmitted from the imageencoder and synchronously performs each decoding process of the baselayer and the enhance layer. When an image is decoded and displayed inreal time, the decoding process, for example, follows the procedureshown in FIG. 10.

In FIG. 10, based upon occupied amount information 116 of a bitstreamdata saved in the receiving buffer 115, a time management function ofthe decoding control part 114 determines a decoding start time of aframe being at the time t. When the time comes (step Silo), the timemanagement function of the decoding control part 114 runs the base-layerdecoding part 117 and the basis decoding part 119 of the enhance-layerdecoding part 118 (steps S111 through S113). While performing a motioncompensation process, the basis-layer decoding part 117 uses the framememory 104 to generate a decoded image 120 in the base layer (stepS112). The basis decoding part 119 analyzes atom data in the bitstreamin accordance with a predetermined syntax rule (step S113). In an atomdecoding process, a basis is identified by the basis code book 108-2similar to that of the image encoder based upon a basis index, and anencoding distortion decoded image 121 is generated based upon theformula (1) (step S114). In order to generate a high quality imageincluding the enhance layer, the base-layer decoded image 120 is addedto the encoding distortion decoded image 121 and the resulting image iswritten in the displaying buffer 122 (step S115). There is a case inwhich the enhance layer is not properly sent to the image decoderdepending on a line condition. In such a case, only the base-layerdecoded image 120 is written in the displaying buffer 122 and isdisplayed. The displaying buffer 122 may is provided to use a portion ofthe base-layer frame memory 104-2. At the displaying time that isprescribed by the bitstream or is determined according to a ruleprescribed in the image decoder, the decoding control part 114 runs thedisplaying buffer 122 and displays the frame being at the time t on thescreen (steps S116 and S117).

In this embodiment of the present invention, the encoded data by the“Matching Pursuits” is used in the enhance layer to form a qualityhierarchical code. As a result, it is possible to efficiently representwithout distortion a visually noticeable pattern such as a blockdistortion generated by the base-layer encoding by means of the DCT.

Regarding FIG. 8, a line 123 may be a network such as the Internet or adata reading circuit for reading and decoding a bitstream from a mediumin which an output result of the image encoder is recorded.

A description will now be given of an image encoder and an image decoderaccording to a third embodiment of the present invention.

In this embodiment, a description will be given of another example of animage encoder and an image decoder to perform a quality hierarchicalencoding and a transmission by using the “Matching Pursuits”. In theimage encoder and the image decoder according to the embodiment, unlikethe second embodiment of the present invention, an encoding distortionsignal to be encoded is divided into a plurality of classes of signalpatterns, and a signal pattern is classified without the use ofadditional information with reference to code parameters in the baselayer. A control part is designed to use a code book aiming at anindividual pattern. As a result, it is possible to perform the “MatchingPursuits” encoding more efficiently.

FIG. 11 shows a structure of the image encoder and the image decoderaccording to the embodiment of the present invention, wherein thoseparts in FIG. 11 corresponding to the parts described in FIG. 8 aredesignated by the same reference numerals.

In FIG. 11, the image encoder uses an enhance-layer encoding part 201whose structure is different from that of the enhance-layer encodingpart 106 shown in FIG. 8. The enhance-layer encoding part 201 comprisesa basis searching part 202, basis code books 203 a-1 and 203 b-1, aswitch SW 204-1 and a basis encoding part 110. Also, the image decoderuses an enhance-layer decoding part 207 whose structure is differentfrom that of the enhance-layer decoding part 118 shown in FIG. 8. Theenhance-layer decoding part 207 comprises a basis decoding part 208,basis code books 203 a-2 and 203 b-2, and a switch SW 204-2.

An encoding process of the above image encoder basically follows theprocess mentioned with respect to the second embodiment of the presentinvention. Namely, after completing the initialization process (ref.S101 in FIG. 9) and the base-layer encoding process (ref. S102 in FIG.9), the encoding process is performed for the encoding distortionsignal, which is to become enhance information. The enhance-layerencoding process is performed as follows.

From a difference between an input picture signal 101 and a decodedimage 105 in the base layer, a base-layer encoding distortion signal 107is generated. An enhance-layer encoding part 201 encodes the base-layerencoding distortion signal 107 based upon the “Matching Pursuits”. Theenhance-layer encoding part 201 of the embodiment has two kinds of basiscode books A and B (referred to as 203 a-1 and 203 b-1, respectively).It is supposed that the basis code book A 203 a-1 specifically aims at ablock distortion arising when the DCT is performed for the base layerand the basis code book B 203 b-1 aims at an application to a signalpattern other than the block distortion. When the block distortioncaused by a DCT quantization noticeably arises in a decoding image inthe base layer, the component corresponding to the distortion alsoarises in the encoding distortion signal 107. The coarser thequantization is, the more noticeably the component arises. In addition,since a position of the boundary between blocks is fixed, it can bedetermined whether or not to use the code book aiming at the blockdistortion based upon a quantization step value in the base layer and asignal position in which a basis is being searched. For theimplementation, the switch SW 204-1 for switching the basis code booksA/B is used. When receiving a quantization step value 206 in thebase-layer and a signal position 205 in which a basis is being searched,the switch SW 204-1 determines which to use the basis code book A or Bin accordance with a predetermined criterion so as to employ a basisvector thereof.

In this case, since the switch SW 204-2 in the image decoder uses anexisting value as determination information, it is unnecessary totransmit some additional information for the determination and it ispossible to efficiently perform the “Matching Pursuits” encoding processdynamically adaptable to the signal pattern.

The basis searching part 202 operates similarly to the basis searchingpart 109 mentioned in the second embodiment except that the basissearching part 202 supplies the signal position in which a basis isbeing searched for to the switch SW 204-1.

The above image decoder uses the enhance-layer decoding part 207 onlywhose structure is different from that of the enhance-layer decodingpart 118 (ref. FIG. 8) in the second embodiment. The basis decoding part208 operates similarly to the basis decoding part 119 mentioned in thesecond embodiment except that the basis decoding part 208 outputs atomposition information 205 to the switch SW 204-2. Also, the switch SW204-2 receives the quantization step value 206 at the correspondingposition in the base layer from the base-layer decoding part 117. Withreference to the atom position information 205 and the quantization stepvalue 206, the switch SW 204-2 determines which basis code book A or Bshould be used in accordance with the same criterion as theenhance-layer encoding part 201.

In the above system structure, it is possible to classify an encodingdistortion signal to be encoded by the “Matching Pursuits” into either aclass of visually noticeable signal patterns such as a block distortioncaused by encoding the base-layer with the use of the DCT or a class ofother signal patterns with respect to the quality hierarchical encodingusing the “Matching Pursuits” encoding. In addition, it is possible touse selectively the basis code book suitably for each class. At the sametime, since the basis code book is segmented before the search, it ispossible to decrease the number of arithmetic operations for searchingan atom and shorten the code length of the basis index. When thebase-layer encoding parameter is used to choose the basis code book,particular information is not required. More than or equal to two basiscode books may be prepared. One basis code book may be divided into someclasses.

In the embodiment, the description has been given of another example ofthe image encoder and the image decoder to perform a qualityhierarchical encoding and a transmission by using the “MatchingPursuits”. Unlike the structure of the second embodiment mentionedabove, the embodiment introduces a motion compensation prediction so asto remove the redundancy in the time direction of the encodingdistortion signal to be encoded. The use of the motion compensationprediction makes it possible to more efficiently encode the enhancelayer.

A description will now be given of an image encoder and an image decoderaccording to a fourth embodiment of the present invention. In thisembodiment, a description will be given of another example of the imageencoder and the image decoder to perform a quality hierarchical encodingand a transmission by using the “Matching Pursuits”. Unlike thestructure of the second embodiment mentioned above, the embodimentintroduces motion compensation prediction so as to remove the redundancyin the time direction of the encoding 43. distortion signal to beencoded. The use of the motion compensation prediction makes it possibleto more efficiently encode the enhance layer.

FIG. 12 shows a structure of the image encoder and the image decoderaccording to the fourth embodiment of the present invention, whereinthose parts in FIG. 12 corresponding to the parts described in FIG. 8are designated by the same reference numerals.

In FIG. 12, the image encoder uses an enhance-layer encoding part 301whose structure is different from that of the enhance-layer encodingpart 106 shown in FIG. 8. The enhance-layer encoding part 301 comprisesa basis searching part 109, a basis encoding part 110, a basis localdecoding part 302, a motion detection and prediction signal generatingpart 303, an enhance-layer frame memory 304-1 and a basis code book308-1. Also, the image decoder uses an enhance-layer decoding part 309whose structure is different from that of the enhance-layer decodingpart 118 shown in FIG. 8. The enhance-layer decoding part 309 comprisesan enhance-layer frame memory 304-2, a basis code book 308-2, a basisdecoding part 310 and a prediction signal generating part 311.

An encoding process of the above image encoder basically follows theprocess mentioned with respect to the second embodiment of the presentinvention. Namely, after completing the initialization process (ref.S101 in FIG. 9) and a base-layer encoding process (ref. S102 in FIG. 9),the encoding process is performed for the encoding distortion signalwhich is to become enhance information. The enhance-layer encodingprocess is performed as follows.

In the enhance-layer encoding part 301, an inter frame motioncompensation prediction is performed for the encoding distortion signal107, and the prediction residual signal 307 is encoded. The encodingdistortion signal 107 depends upon an image pattern in the base layer.Since correlation between frames with respect to the encoding distortionsignal can be also considered to be high, the execution of the motioncompensation prediction makes it possible to decrease the redundancy inthe time direction and perform an efficient encoding.

The enhance-layer encoding part 301 encodes the encoding distortionsignal 107 based upon the “Matching Pursuits” with the motioncompensation. FIG. 13 shows a procedure of the motion compensationprediction.

When the encoding distortion signal 107 is delivered to the motiondetection and prediction signal generating part 303, a prediction signal305 is generated. The prediction signal 305 is determined permacroblock. The macroblock counter n is reset (step S301). Then, theintra frame prediction is performed for the macroblock n (step S302). Avariety of intra frame prediction methods is assumed. For example, apixel average of an input macroblock (DC component) may be used as aprediction image. Also, values of marginal pixels of the macroblock thathave already been encoded may be buffered, and the values may be used togenerate a prediction image through a plane prediction.

In step S302, a prediction cost incurred by the prediction image data(P_intra) and the inter frame prediction is computed. When a pluralityof prediction methods are used, intra frame prediction mode informationis also produced to indicate what type of intra frame prediction methodhas been used. Regarding the prediction cost, a sum of absolutedifference (SAD) and a mean square error (MSE) may be used to representa residual power. Besides, a code amount-distortion cost representationincluding additional information required for a prediction method (suchas above-mentioned intra frame prediction mode information) may be used.

Then, the inter frame prediction is performed through a motioncompensation prediction (step S303). The inter frame prediction predictsa motion between frames and identifies a similar image in referenceimages in the enhance-layer frame memory 304-1. In motion detection, amotion vector is detected as information to identify the similar imagein the reference images. The motion vector may be represented in anyform such as a parallel shift vector by block matching often used inexisting picture encoding methods, a vector representing an affinemodel, or a vector representing a perspective transform model. Ingeneral, the more complicated a model is and the larger the number ofmotions that can be represented, the smaller the prediction residual is.On the other hand, since arithmetic operations for the motion detectionand the number of parameters for the model representation increasecorresponding to the above benefit, it is necessary to make a selectionfrom the viewpoint of a balance between implementation burden andperformance. Regarding motion vector search and representation method,any representation is available in this embodiment of the presentinvention as long as it can be represented as motion informationrepresentation having a macroblock as a unit. Also, like an 8×8 pixelunit prediction mode of MPEG-4, a macroblock may be further divided intosmaller blocks and a motion vector is found for each divided block. Likea B frame prediction of MPEG-½, a plurality of frames may be referred toin order to perform a prediction. In a case in which a plurality ofprediction methods are switched, inter frame prediction mode informationis produced to indicate what type of prediction method has been used.The inter frame prediction mode may be such a mode that “MatchingPursuits” encoding data of an encoded frame is saved (for the storage,it is required to prepare a memory space to save an atom parameter 24corresponding to the frame memory 11) and atom information encoded in areference frame is inherited without change. At step S303, predictionimage data (P_inter), a motion vector, inter frame prediction modeinformation, and a prediction cost incurred by a motion compensationprediction are computed. Regarding the prediction cost, the SAD, the MSEand the code amount-distortion cost may be used like the intra frameprediction. As a code amount which should be considered in the interframe prediction, there are a code amount of a motion vector itself,switch information of a motion vector model, inter frame prediction modeinformation and so on.

At step S304, it is determined which type of prediction is performed forthe macroblock of interest, the intra frame prediction (INTRA) or theinter frame prediction (INTER). The two above-mentioned prediction costsare used as an evaluation value with respect to the predictions and theprediction incurring the smaller prediction cost is chosen. If the intraframe prediction (INTRA) is chosen, a prediction image Pred(n) for themacroblock n is set to a P_intra and the intra frame prediction modeinformation is output as output information 306. If the inter frameprediction (INTER) is chosen, a prediction image Pred(n) is set to aP_inter and the inter frame prediction mode information together with amotion vector are output as the output information 306 (steps S305 andS306). After that, the macroblock counter n is incremented (S307). Ifthe above-mentioned process is completed for every macroblock in theframe (step S308), the prediction image generating process is completed.

A difference between the generated prediction image 305 and the encodingdistortion signal 107 is computed. Then, the computed residual signal307 is supplied to the basis searching part 109. Since the operations ofthe basis searching part 109 and the basis encoding part 110 are thesame as that of the first embodiment of the present invention, thedescription thereof will be omitted. The basis code book 308-1 isdesigned to reflect the prediction residual pattern of the encodingdistortion signal in the time direction.

Although not shown in the diagram, a plurality of prepared basis codebooks may be switched according to the quantization step value in thebase layer or atom position information similarly to the thirdembodiment. Especially regarding the atom position information, adiscontinuous portion between macroblocks may arise in a predictionimage because the motion detection by the enhance-layer encoding part301 has a macroblock as a unit. Thus, it is possible to efficientlyperform the “Matching Pursuits” if the code book suitable for thediscontinuous signal pattern is prepared.

Motion vector and prediction mode information 306 together with an atomparameter 313 are multiplexed to a bitstream and are transmitted to theimage decoder. The encoded atom parameter 313 is sent to the basis localdecoding part 302 and a decoded image of the prediction residual signal307 is generated. The decoded image is saved in the enhance-layer framememory 304-1 for encoding the next frame.

In the above image decoder, only the structure of the enhance-layerdecoding part 309 is different from that of the enhance-layer decodingpart 118 in the second embodiment (ref. FIG. 8). In the enhance-layerdecoding part 309, the basis decoding part 310 performs the sameoperations as the basis local decoding part 302 with reference to thebasis code book 308-2. The prediction signal generating part 311generates a prediction image 305 according to the motion vector andprediction mode information 306. The decoding control part 114 controlsoperations of each decoding part so that frames of the base layer andthe enhance layer can be decoded synchronously. Other operations such asthe displaying control follow the operation mentioned in the secondembodiment.

According to the structure of the embodiment, in a quality hierarchicalencoding using the “Matching Pursuits”, if the motion compensationprediction is introduced for the encoding distortion signal to beencoded on the “Matching Pursuits”, it is possible to more efficientlyencode the enhance layer. The above-mentioned enhance-layer encodingpart 301 and the enhance-layer decoding part 309 may be substituted foran enhance-layer encoding part 410 and an enhance-layer decoding part405, respectively.

In this case, in the enhance-layer encoding part motion vectorinformation obtained as an encoding result of the base layer is used asa motion vector of the macroblock which is located at the same positionas the enhance layer. In order to simplify a process of a motiondetection and prediction signal generating part 403 and decreaseadditional information to be transmitted, the enhance-layer encodingpart 401 uses the motion vector of the macroblock located at the sameposition as the enhance layer which has been detected in the base layerto generate an inter frame prediction signal. In general, since acorrelation of the image pattern between the base layer and the enhancelayer is high in the quality hierarchical encoding, motion informationdetected in the base layer is often directly used in the enhance layerefficiently. Also, in order to increase the coding efficiency in theenhance layer, motion detection may use the base-layer motioninformation 402 as an initial value to be performed for an infinitesimaldomain around the base-layer motion information 402. Since initial valueregarding a motion searching point is given, it is sufficient to searcha motion only in the neighborhood of the initial value and it ispossible to reduce the number of arithmetic operations. In this case, adetection result in the enhance layer is sent to the image decoder asadditional information, which is set as difference information 404 basedupon the base-layer motion information 402. If the base-layer motioninformation 402 is directly used as the motion difference information404, it is unnecessary to transmit the motion difference information404.

In such a case in which the base-layer motion vector information is usedto perform the enhance-layer motion compensation prediction in theenhance-layer encoding part 401, the enhance-layer decoding part 405receives the base-layer motion information 402 from the base-layerdecoding part 117 and uses the motion difference information 404included in a bitstream in the enhance layer to generate the predictionimage 305 in a prediction signal generating part 406.

INDUSTRIAL APPLICABILITY

As mentioned above, according to the claims 1 through 62, it is possibleto realize the image encoding method and the image encoder and thecorresponding image decoding method and the corresponding image decodercapable of improving the total coding efficiency by representing encodeddata more compactly by means of the pattern matching technique. Also,according to the claims 63 and 64, it is possible to realize the imageprocessing system that performs an image encoding and decoding inaccordance with such a method mentioned in the previous claims.

1. An image encoding method, comprising: a first image encoding stepperforming a compression encoding for a first image signal perpredetermined partial domain in accordance with a first encoding rule;and a second image encoding step performing a compression encoding for asecond image signal per partial signal waveform of the second imagesignal in accordance with a second encoding rule, wherein a signalcorresponding to an error signal between a first image signal providedto the first encoding step and a signal obtained by performing a localdecoding process for encoding information obtained at the first imageencoding step is set as a second image signal provided to the secondimage encoding step, and wherein the second image encoding stepcomprises a pattern matching step searching for a waveform vector in apredetermined group of waveform vectors based upon inner products of thewaveform vectors and a partial signal waveform of the second imagesignal; and a waveform pattern encoding step performing a compressionencoding for information to identify the waveform vector searched for,an inner product value of the partial signal waveform and the waveformvector searched for, and a position in the second image of the partialsignal waveform in accordance with the second encoding rule, wherein ina process in which the pattern matching step and the waveform patternencoding step are iterated until a predetermined condition is satisfied,whenever encoding information is obtained at the waveform patternencoding step, a reconstructed waveform of a partial signal waveformobtained from the encoding information is subtracted from the secondimage signal so as to produce a second image signal to be provided tothe pattern matching step, and the second image signal is encodedaccording to the encoding information obtained at the waveform patternstep until the predetermined condition is satisfied, and wherein theposition in the image of the partial signal waveform is encoded throughreplacement with position information in a predetermined partial domainof the image signal and in the pattern matching step, a group ofwaveform vectors to be used is selected among a plurality of groups ofwaveform vectors based on a parameter used to perform the compressionencoding at the first image encoding step and the position in the secondimage of the partial waveform signal.
 2. The image encoding method asclaimed in claim 1, wherein the first image signal to be provided to thefirst image encoding step becomes information representing a sourceimage to be encoded, and the second image signal to be provided to thesecond image encoding step becomes an encoding distortion signalgenerated by encoding the source image at the first image encoding step.3. The image encoding method as claimed in claim 2, wherein the secondimage signal becomes information representing a prediction residualimage obtained by performing a motion compensation prediction for theerror signal.
 4. The image encoding method as claimed in claim 3,wherein at the second image encoding step, when the informationrepresenting the prediction residual image is obtained, motioninformation detected in an encoding process for a source image at thefirst image encoding step is used to perform the motion compensationprediction.
 5. The image encoding method as claimed in claim 3, whereinat the second image encoding step, the motion information detected in anencoding process for a source image at the first image encoding step isset as an initial value, motion information is detected in positionsadjacent to a position where a motion regarding the initial value hasoccurred, and the motion information is used to perform the motioncompensation prediction.
 6. An image encoder, comprising: a first imageencoding part performing a compression encoding for a first image signalper predetermined partial domain in accordance with a first encodingrule; and a second image encoding part performing a compression encodingfor a second image signal per partial signal waveform of the secondimage signal in accordance with a second encoding rule, wherein a signalcorresponding to an error signal between a first image signal providedto the first encoding part and a signal obtained by performing a localdecoding process for encoding information obtained by the first imageencoding part is set as a second image signal provided to the secondimage encoding part, and wherein the second image encoding partcomprises a pattern matching part searching for a waveform vector in apredetermined group of waveform vectors based upon inner products of thewaveform vectors and a partial signal waveform of the second imagesignal; and a waveform pattern encoding part performing a compressionencoding for information to identify the waveform vector searched for,an inner product value of the partial signal waveform and the waveformvector searched for, and a position in the second image of the partialsignal waveform in accordance with the second encoding rule, wherein ina process in which operations on the pattern matching part and thewaveform pattern encoding part are iterated until a predeterminedcondition is satisfied, whenever encoding information is obtained by thewaveform pattern encoding part, a reconstructed waveform of a partialsignal waveform obtained from the encoding information is subtractedfrom the second image signal so as to produce a second image signal tobe provided to the pattern matching part, and the second image signal isencoded according to the encoding information obtained by the waveformpattern part until the predetermined condition is satisfied, and whereinthe pattern matching part comprises a plurality of groups of waveformvectors; and a waveform vector group selection part selecting a group ofwaveform vectors to be used among the groups of waveform vectors basedupon a parameter used by the first image encoding part to perform thecompression encoding and the position in the second image of the partialwaveform signal.
 7. The image encoder as claimed in claim 6, wherein thefirst image signal to be provided to the first image encoding partbecomes information representing a source image to be encoded, and thesecond image signal to be provided to the second image encoding partbecomes an encoding distortion signal generated by encoding the sourceimage at the first image encoding part.
 8. The image encoder as claimedin claim 7, wherein the second image signal becomes informationrepresenting a prediction residual image obtained by performing a motioncompensation prediction for the error signal.
 9. The image encoder asclaimed in claim 8, wherein when the information representing predictionresidual image is obtained, the second image encoding part uses motioninformation detected in an encoding process for a source image at thefirst image encoding part to perform the motion compensation prediction.10. The image encoder as claimed in claim 9, wherein the second imageencoding part sets the motion information detected in an encodingprocess for a source image at the first image encoding part as aninitial value, detects motion information in positions adjacent to aposition where a motion regarding the initial value has occurred, anduses the motion information to perform the motion compensationprediction.
 11. An image processing system, comprising: an image encoderhaving a first image encoding part performing a compression encoding fora first image signal per predetermined partial domain in accordance witha first encoding rule; and a second image encoding part performing acompression encoding for a second image signal per partial signalwaveform of the second image signal in accordance with a second encodingrule, wherein a signal corresponding to an error signal between thefirst image signal provided to the first encoding part and a signalobtained by performing a local decoding process for encoding informationobtained by the first image encoding part is set as the second imagesignal provided to the second image encoding part, and wherein thesecond image encoding part comprises: a pattern matching part searchingfor a waveform vector in a predetermined group of waveform vectors basedupon inner products of the waveform vectors and a partial signalwaveform of the second image signal; and a waveform pattern encodingpart performing a compression encoding for information to identify thewaveform vector searched for, an inner product value of the partialsignal waveform and the waveform vector searched for, and a position inthe second image of the partial signal waveform in accordance with thesecond encoding rule, wherein in a process in which operations on thepattern matching part and the waveform pattern encoding part areiterated until a predetermined condition is satisfied, whenever encodinginformation is obtained at the waveform pattern encoding part, areconstructed waveform of a partial signal waveform obtained from theencoding information is subtracted from the second image signal so as toproduce a second image signal to be provided to the pattern matchingpart, and the second image signal is encoded according to the encodinginformation obtained at the waveform pattern part until thepredetermined condition is satisfied, and wherein the pattern matchingpart comprises: a plurality of groups of waveform vectors; and awaveform vector group selection part selecting a group of waveformvectors to be used among the groups of waveform vectors based upon aparameter for the encoding at the first image encoding part and theposition in the second image of the partial waveform signal; and animage decoder comprising: a first image decoding part receiving a firstcompressed image and reconstructing first image information bydecompressing the first compressed image information per predeterminedpartial domain; a second image decoding part receiving a secondcompressed image and reconstructing second image information bydecompressing the second compressed image information per predeterminedpartial signal waveform; and an image synthesizing part obtaining outputimage information by synthesizing the first image information and thesecond image information, wherein the second image decoding partcomprises: a plurality of predetermined groups of waveform vectors; afirst part decoding the second compressed image information inaccordance with a predetermined decoding rule and obtaining informationto identify a waveform vector, an inner product value of a partialsignal waveform and the waveform vector and a position in an image ofthe partial signal waveform; a second part selecting a group of waveformvectors to be used among a plurality of predetermined groups of waveformvectors based upon a code parameter included in the first compressedimage information provided to the first image decoding part and theposition in the image of the partial waveform signal; and a third partgenerating the second image information based upon the waveform vectoridentified by the information to identify the waveform vector in theselected group of waveform vectors, the inner product value, and theposition in the image of the partial signal waveform.